GH-46179: [Python] Bump index level once if pandas df already contains __index_level_i__ column - #46884

Pull request overview

This PR addresses GH-46179 in PyArrow’s pandas conversion by avoiding duplicate Arrow field names when a pandas DataFrame already contains __index_level_i__ columns, ensuring generated index columns use a non-conflicting name.

Changes:

• Update generated index column naming to pick the next available __index_level_{j}__ name if the default collides with existing columns.
• Ensure uniqueness across both DataFrame columns and previously generated index columns when multiple index levels are serialized.
• Add regression tests for single-index and MultiIndex cases where __index_level_0__ already exists as a DataFrame column.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

To be fair I don't know much about the __index_level_{i} works.
Can we add an explicit test to validate the index is respected?

Based on the issue snippet:

In [40]: df = pd.DataFrame({"col": [1, 2, 3], "__index_level_0__": [1, 2, 3]}, index=[2, 3, 4])

In [41]: df
Out[41]: 
   col  __index_level_0__
2    1                  1
3    2                  2
4    3                  3

In [42]: pa.table(df)
Out[42]: 
pyarrow.Table
col: int64
__index_level_0__: int64
__index_level_0__: int64
----
col: [[1,2,3]]
__index_level_0__: [[1,2,3]]
__index_level_0__: [[2,3,4]]

I would expect the pa.table(df).to_pandas() to continue presenting the index correctly:

>> pa.table(df).to_pandas()
   col  __index_level_0__
2    1                  1
3    2                  2
4    3                  3

Can we validate the pandas index is the new __index_level_1__ instead of the column __index_level_0__?